Tree-based Fitted Q-iteration for Multi-Objective Markov Decision Processes in Water Resources Management

Multi–Objective Markov Decision Processes (MOMDPs) provide an effective modeling framework for multi–objective decision-making problems involving water resources systems. The traditional approach to solve these problems is to consider many single–objective problems (resulting from different combinations of the original problem objectives), each one solved using standard optimization techniques. This paper presents a new approach to MOMDPs based on batch–mode reinforcement– learning (RL) that enables to learn the operating policies for all the linear combinations of weights assigned to the objectives in a single training process. The key idea is to enlarge the continuous approximation of the action–value function, which is performed by single–objective RL algorithms over the state–action space, also to the weight space. The batch–mode nature of the algorithm makes it possible to enrich the training data without further interaction with the controlled system. The approach is first demonstrated by application to a numerical Test case study of a two–objective reservoir and then evaluated on a real world case study concerning the optimal operation of the Hoa Binh water reservoir in Vietnam. Experimental results on the Test case show that the proposed approach (named MOFQI) becomes com-